We propose Universal Document Processing (UDOP), a foundation Document AI model which unifies text, image, and layout modalities together with varied task formats, including document understanding and generation. UDOP leverages the spatial correlation between textual content and document image to model image, text, and layout modalities with one uniform representation. With a novel Vision-Text-Layout Transformer, UDOP unifies pretraining and multi-domain downstream tasks into a prompt-based sequence generation scheme. UDOP is pretrained on both large-scale unlabeled document corpora using innovative self-supervised objectives and diverse labeled data. UDOP also learns to generate document images from text and layout modalities via masked image reconstruction. To the best of our knowledge, this is the first time in the field of document AI that one model simultaneously achieves high-quality neural document editing and content customization. Our method sets the state-of-the-art on 9 Document AI tasks, e.g., document understanding and QA, across diverse data domains like finance reports, academic papers, and websites. UDOP ranks first on the leaderboard of the Document Understanding Benchmark (DUE).

尽管在文档理解方面取得了成功，但由于计算中的几个挑战以及如何有效吸收长期多模式输入，因此长期文档理解的实际任务在很大程度上尚未探索。大多数基于变压器的方法仅处理简短的文档，并且由于其过度的计算和内存限制，因此仅使用文本信息来引起注意。为了解决长期文档理解中的这些问题，我们探索了处理1D和新的2D位置引人入胜的不同方法，并以本质上的背景缩短了。实验结果表明，我们提出的模型基于各种评估指标具有此任务的优势。此外，我们的模型仅对注意力进行更改，因此很容易适应任何基于变压器的体系结构。

图像变压器最近使用监督（VIT，DEIT等）或自我监督（BEIT，MAE等）预训练技术取得了显着的自然图像理解进展。在本文中，我们提出了\ textbf {dit}，一种自我保护的预训练\ textbf {d} ocument \ textbf {i} mage \ textbf {t} ransformer模型，使用大规模的不尺度的文本图像用于文档AI任务，这是必不可少的，因为由于缺乏人类标记的文档图像，因此没有受到监督的同行。我们将DIT作为骨干网络在各种基于视觉的文档AI任务中，包括文档图像分类，文档布局分析，表检测以及OCR的文本检测。实验结果表明，自我监管的预训练的DIT模型可在这些下游任务上实现新的最新结果，例如文档图像分类（91.11 $ \ rightarrow $ 92.69），文档布局分析（91.0 $ \ rightArow $ 94.9），表检测（94.23 $ \ rightArrow $ 96.55）和OCR的文本检测（93.07 $ \ rightarrow $ 94.29）。代码和预培训模型可在\ url {https://aka.ms/msdit}上公开获得。

我们研究识别结构化文本的问题，即跟随某些格式的文本，并建议通过指定偏见的正则表达式（正则表达）来提高结构化文本的识别准确性。偏置识别器识别与在其他文本上一般小的降级的成本中具有显着提高的指定正则表达式的文本。通过将正数表达式作为加权有限状态换能器（WFST）建模并通过动态替换将其注入解码器中来实现偏置。单个纤维率控制偏置强度。该方法可用于识别具有已知格式的文本行或包含来自域词汇的单词。示例包括驾驶执照号码，处方药中的药物名称等。我们展示了Regex偏置在印刷和手写的结构文本数据集上的功效，并测量其副作用。

文本识别是文档数字化的长期研究问题。现有的方法通常是基于CNN构建的，以用于图像理解，并为Char-Level文本生成而建立RNN。此外，通常需要另一种语言模型来提高整体准确性作为后处理步骤。在本文中，我们提出了一种使用预训练的图像变压器和文本变压器模型（即Trocr）提出的端到端文本识别方法，该模型利用了变压器体系结构，以实现图像理解和文字级级文本生成。TROR模型很简单，但有效，可以通过大规模合成数据进行预训练，并通过人体标记的数据集进行微调。实验表明，TROR模型的表现优于印刷，手写和场景文本识别任务上的当前最新模型。Trocr模型和代码可在\ url {https://aka.ms/trocr}上公开获得。

由于其有效的模型架构以及大规模未标记的扫描/数字出生的文件的优势，在各种视觉上丰富的文档理解任务中已经证明了文本和布局的预先培训。我们提出了具有新的预培训任务的Layoutlmv2架构，以在单个多模态框架中模拟文本，布局和图像之间的交互。具体地，对于双流多模态变压器编码器，LayOutLMV2不仅使用现有屏蔽的视觉语言建模任务，还使用新的文本图像对齐和文本图像匹配任务，这使得它更好地捕获跨模块交互在预训练阶段。同时，它还将空间感知的自我注意机制集成到变压器架构中，以便模型可以完全理解不同文本块之间的相对位置关系。实验结果表明，LayoutLMV2优于大幅度的LayOutlm，并在大量下游的下游富有的文件理解任务中实现了新的最先进的结果，包括Funsd（0.7895 $ \至0.8420美元），电源线（0.9493 $ \至0.9601美元），Srie（0.9524 $ \至0.9781美元），Kleister-NDA（0.8340 $ \ 0.8520美元），RVL-CDIP（0.9443 $ \至0.9564美元），DOCVQA（0.7295 $ \至0.8672美元） 。我们使我们的模型和代码公开可用于\ url {https://aka.ms/layoutlmv2}。

Harmonic functions are abundant in nature, appearing in limiting cases of Maxwell's, Navier-Stokes equations, the heat and the wave equation. Consequently, there are many applications of harmonic functions, spanning applications from industrial process optimisation to robotic path planning and the calculation of first exit times of random walks. Despite their ubiquity and relevance, there have been few attempts to develop effective means of representing harmonic functions in the context of machine learning architectures, either in machine learning on classical computers, or in the nascent field of quantum machine learning. Architectures which impose or encourage an inductive bias towards harmonic functions would facilitate data-driven modelling and the solution of inverse problems in a range of applications. For classical neural networks, it has already been established how leveraging inductive biases can in general lead to improved performance of learning algorithms. The introduction of such inductive biases within a quantum machine learning setting is instead still in its nascent stages. In this work, we derive exactly-harmonic (conventional- and quantum-) neural networks in two dimensions for simply-connected domains by leveraging the characteristics of holomorphic complex functions. We then demonstrate how these can be approximately extended to multiply-connected two-dimensional domains using techniques inspired by domain decomposition in physics-informed neural networks. We further provide architectures and training protocols to effectively impose approximately harmonic constraints in three dimensions and higher, and as a corollary we report divergence-free network architectures in arbitrary dimensions. Our approaches are demonstrated with applications to heat transfer, electrostatics and robot navigation, with comparisons to physics-informed neural networks included.

We tackle open-world semantic segmentation, which aims at learning to segment arbitrary visual concepts in images, by using only image-text pairs without dense annotations. Existing open-world segmentation methods have shown impressive advances by employing contrastive learning (CL) to learn diverse visual concepts and adapting the learned image-level understanding to the segmentation task. However, these methods based on CL have a discrepancy since it only considers image-text level alignment in training time, while the segmentation task requires region-text level alignment at test time. In this paper, we propose a novel Text-grounded Contrastive Learning (TCL) framework to directly align a text and a region described by the text to address the train-test discrepancy. Our method generates a segmentation mask associated with a given text, extracts grounded image embedding from the masked region, and aligns it with text embedding via TCL. The framework addresses the discrepancy by letting the model learn region-text level alignment instead of image-text level alignment and encourages the model to directly improve the quality of generated segmentation masks. In addition, for a rigorous and fair comparison, we present a unified evaluation protocol with widely used 8 semantic segmentation datasets. TCL achieves state-of-the-art zero-shot segmentation performance with large margins in all datasets. Code is available at https://github.com/kakaobrain/tcl.

We present HOReeNet, which tackles the novel task of manipulating images involving hands, objects, and their interactions. Especially, we are interested in transferring objects of source images to target images and manipulating 3D hand postures to tightly grasp the transferred objects. Furthermore, the manipulation needs to be reflected in the 2D image space. In our reenactment scenario involving hand-object interactions, 3D reconstruction becomes essential as 3D contact reasoning between hands and objects is required to achieve a tight grasp. At the same time, to obtain high-quality 2D images from 3D space, well-designed 3D-to-2D projection and image refinement are required. Our HOReeNet is the first fully differentiable framework proposed for such a task. On hand-object interaction datasets, we compared our HOReeNet to the conventional image translation algorithms and reenactment algorithm. We demonstrated that our approach could achieved the state-of-the-art on the proposed task.

Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for language models has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply performing gradient ascent on target token sequences is effective at forgetting them with little to no degradation of general language modeling performances for larger LMs; it sometimes even substantially improves the underlying LM with just a few iterations. We also find that sequential unlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with a previous data preprocessing method and a decoding method known to mitigate privacy risks for LMs, we show that unlearning can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust. We release the code and dataset needed to replicate our results at https://github.com/joeljang/knowledge-unlearning.